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ABSTRACT 

The reading comprehension items for the Nelson 
Reading Skills Test, a revision of a widely used standardized reading 
testr were administered to several hundred fourth- and sixth-grade 
students in order to determine whether the student^s ability to 
answer correctly actually depended on his comprehension of the 
accompanying passage. All the field-tested items were divided into 
nine booklets, which contained one normetl reading comprehension 
subtest and one subtest with only the questions. An index of passage 
dependency was computed for each item, and the index was used to help 
the authors locate items that were weak. The index also indicated the 
effect of passage dependency on skill categories. (Author/BW) 
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OF READING COMPREHENSION TEST ITEMS 



Most standardized reading comprehension tests have several sets of one or 
more multiple-choice questions which follow a sentence, paragraph, or passage 
of reading material. Supposedly, the number of questions answered correctly Is 
a measure of how well the student understands the reading material. However, 
the research evidence Indicates that this Is not always the case. 

Various writers have noted that unless the passages and items are carefully 
Mxltten, it may be possible for students to answer the questions correctly with- 
Dut reading the passages (Weaver, Bickley, and Ford, 1969; Pyrczak, 1974; Pyrczak, 
1975; and Tulnman, 1974). In these studies students were able to score consid- 
irably above chance levels on various reading tests without reading the passages. 
Apparently, the students determine correct responses by obtaining cues from other 
luestlcns in the set, by eliminating faulty and irrelevant distractors, and from 
general knowledge about the topic. 

Tulnman (1970) administered the first 40 items of the Sequential Test of 
Mucational Progress-Reading , Form 3A, to 134 seventh, eighth, and ninth graders 
XL two conditions—reading passages and without reading passages. The mean score 
or the without passages test was 34 per cent of the possible score, and it was 
8 per cent with passages. 

Farr and Smith (1970) removed the paragraphs from the Nelson- Denny com- 
rehenslon test and administered it to a group of college sophomores; then, three 
eeks later re-administered it to them with the paragraphs Included. They found 
hat for 5 of the 32 items, the number of correct responses under the passage«-out 



3 




- 2 - ' 

condition exceeded the number of correct responses under the passage- in condi- 
tion. Also, for 12 of the items, the number of correct responses in the passage- 
out condirion exceeded 50 per cent. 

Tulnman (1974) studied the pass2ge dependency of five major reading tests. 
Using a sample of 9,451 fourth, fifth, and sixth grade students, he found that 
the students' performance without passages was considerably above chance score 
levels. In fact, when students were not allowed to read the passages, their 
performance loss was less than 30 per cent of their passage-in performance. 
This study also showed that on these tests passage dependency as a more seri- 
ous problem in gtade 4 than in grades 5 and 6. 

Tuinman (1974) points out that finding low passage dependency in a study 
in which the items are administered without the reading passages may not nec- 
essarily invalidate the test. Jn normal testing situations the same conditions 
as a research study would not result unless the tudent simply skipped the 
reading material and tried to answer the questions. However, if the reading 
material is skipped, the test is not valid as a reading measure. Therefore, 
a research study showing low passage dependency signals a potential of in- 
validity more than actual lack of validity. 

Both Tuinman (1974) and Pyrczak (1974, 1975) indicate that teachers, 
school administrators, and test publishers should give more attention to the 
passage dependency situation of reading comprehension tests. Vyvczak (1974) 
suggested : 

... in future test-development projects, reading-comprehension 
items routinely be administered in the absence of the associated 
passages to determine their passage-dependence. Such a procedure 
should contribute to the measurement of true-score variance asso- 
ciated with reading- comprehension abilities and thus to the en- 
hancement of the validity of a test of reading comprehension, 
(p. 347) 
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Pyrczak*s suggestion seems very appropriate. As previously Indicated » 
low passage~dependency may not Invalidate a test, but Increasing the passage 
dependency seems likely to Improve It a measure of reading comprehension. 
It seems logical that the more cues, faulty dlstractors, and general knowledge 
questions that can be eliminated , the more likely the scores will Indicate 
true reading ability. Passage-dependency tryouts of Items should help reveal 
problems with the Items. 
This Study 

The co-author of this paper and Houghton Mifflin Company have developed 
a new reading test, the Nelson P^adlng Skills Test (NRST) , which Is designed 
for grades 3 through 9. During the NRST*s development a passage-dependency 
study of the reading comprehension subtest Items was completed. That study, 
Its results and Implications will be discussed In the remainder of this paper. 

The passage-dependency tests \;ere administered In October, 1975. The 
sample consisted of approximately 1600 Grade four and Grade six students 
from five widely separated school districts. On an Index of socio-economic 
status (SES) which was composed from Information about family Income and edu- 
cation for all school districts In the United States, the weighted average 
of SES*s for these five districts was slightly below the mld-polnt. A few 
of the answer sheets were unscorable so the analysis was completed on 761 
and 813 answer sheets from Grades 4 and 6, respectively. 

A total of 327 Items was used in the passage-dependency study. The 
items were divided among nine test booklets. The first subtest of each book- 
let contained a normal reading comprehension test with reading passage and 
questions about the passage; the second subtest contained approximately an 
equal number of questions but without reading passages. The large number 
of items and booklets made it possible to arrange each booklet so that the 
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questions in the passage-out subtest were not related Lo any passages in the 
regular reading comprehension subtest. 

Every question appeared twice: once, with its reading passage arid once 
in another booklet, without its reading passage. In the passage-out subtest, 
sets of questions about a single passage were kept together as a unit. 

The administration of test booklets by grades was as follows: 

Grade Booklets Total xtems 

4 1,2,3,4,5,6 218 

6 3, ,5,6,7,8,9 : 

The 74 items in booklets 1 and 2 were deemed to be too easy for Grade 6 
and the 109 items in booklets 7, 8 and 9 were deemed too difficult for Grade 
4. The 144 items in booklets 3, 4, 5 and 6 had been developed for students 
in the entire middle grade range; thus, they were administered to students 
in both grades. 

The directions for passage-out subtest, which the teacher read aloud to 
the students as they silently read them, explained that the students were 
taking a special test. These directions indicated that students would 
normally read some material and answer questions about what they had read, 
but this test was trying to determine how many questions the students might 
be able to answer without reading the material. They were instructed to 
read each question carefully because they might be able to find a hint 
about the correct answer in the question. Appendix A contains a reproduction 
of the instruction page. 

Each subtest had a 25-minute time limit. In earlier tryouts and in this 
study, over 95% of the students finished the test. Each student took one 
test booklet. The booklets were handed out in a rotated order in each class- 
room. All items were multiple choice with four options. 
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The number of answer sheets analyzed for each test booklet and grade Is 
shown in Table I. 



Table I 

NUMBER OF STUDENTS TAKING EACH TEST BOOKLET 



Test Booklet Grade 4 Grade 6 

1 125 

2 126 

3 135 118 

4 131 124 

5 125 121 

6 119 120 

7 111 

8 109 

9 110 



The numbers of items tried out in each grade of this study are shown in 
Table II. Two kinds of items were tried out in the study, reading comprehension 
and words in context; they were analyzed separately. 

Table II 

NUMBER OF ITEMS PER GRADE 

Grade 4 Grade 6 

Reading Comprehension 176 192 

Words in Context 42 61 

Total 218 253 

Results 

The item data from the tryout were analyzed by comparing three statistics. 
First, the percentage of students selecting the correct answer with the passage- 
in (PI). Second, the percentage of students selecting the correct answer with 
the passage-out (PO). Third, these percentages were used to .ompute a pasoage- 
dependenc]^ index (PDI) as suggested by Tuinman (1974) : 
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PDI - 1 - dp^^/dpj 



Where: 

Note: Tulnman's formula used NP to represent the passage-out condition 

and P to represent the passage-in condition. We believe PO and 
PI is less confusing. 

The median values for each statistic for all the items in the jtudy are 
shown in Table III. The range of each statistic is shown in Table IV. 



the proportion of „correct_, responses to item i under 
the fassage-out (POJ condition. 

the proportion of'.correct responses to item i under 
the passage-in QPIJ condition. 



Table III 

MEDIAN VALUES FOR PASSAGE-DEPENDENCY STUDY ITEMS 
Reading Comprehension 

Grade 4 (176 Items) Grade 6 (196 Items) 

PO .33 .35 

Pt .60 .65 

PDI .35 .42 

Words in ^[Jontext 

Grade 4 (42 Items) Grade 6 (61 Items) 

PO .32 .41 

PI .48 .63 

PDI .32 ,30 
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Table IV 



RANGE OF EACH STATISTIC FOR PASSAGE-DEPENDENCY STUDY ITEMS 

Reading Comprehension 

Grade 4 (176 Items) Grade 6 (196 Items) 



PC .06 - .80 .10 - ,78 

PI .17 - .95 .17 ~ .97 

PDI -.41 - .93 -.88 - .87** 



Woi'ds in Context 

Grade 4 (42 Items) Grade 6 (61 Items) 

PC .13 - .70 .13 - .87 

PI .13 - .72 .16 - .88 

PDI -.36 - .75 -.38 - .76 



** One item in this group had a PO that was more than double its PI. 

Discussion 

Generally, the results of the passage-dependency study supported the 
validity of the NRST items. While the PO medians were not as low as chance 
score (.25) they were, except for the Words in Context items in Grade six, 
relatively close. More importantly, however, the passage-dependency results 
helped in the selection and/or changing of items in ways believed to be . 
beneficial in constructing a test with greater validity. 

For example, one eight-item selection about skyscrapers, which might have 
been included in the NRST, was clearly shown as very weak by the PDI data. 
The normal kind of item data (i.e. percentage correct and discrimination 
analysis index) would have shown that the items were difficult, but fairly 
good discriminators. However, the median PDI value for the items was 
negative; therefore, it seemed that the items were not only difficult, but 
they were also confusing to th students. 
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Another set of eight-items that w^ns from a selection about sound patterns 
wa6 administered to both grades in the study* The items* median values were 
.28, ,28 and -,09 for PO, PI and PDI, respectively, for Grade four, but were 
.34, .65 and .38 for Grade six. The items also met other selection criteria 
and are included in the level of the test which is appropriate for Grade six 
students. The items from a selection about bullfighters also had a negative 
PDI in Grade four, but it was an acceptable selection for the upper levels. 
Several other selections had similar data but less pronounced differences. 

The data were useful in looking at individual items, too. Several items 
that had low PDI's were slightly altered before being used In the NRST. 
Also, in some selections, items which had low PDI's were dropped. 
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R'iADING COMPREHENSION' - Special Test 



Directions^ This test is like the one you just finished, except the 
reading passages have been rerr.oved. All of the questions arc about reading 
passages, but you will not get to read the passages* This is being done in 
order to determine how many questions you know without doing the receding. 
You will be able -to figure out the answer for many of the questions, but on 
some of them you will simply have to gue-s* 

Please read each question and pick the one best answer for it. On 
your answer sheet, find the row of answer ovals that has the same nuiuber 
as tne question. Fill in the oval that has the same letter as the an-.wer 
you choose. 

Look at the two sample questions below^ 



51. Final means 

A. first 

B. last 

C. long 

D. backward 



52; This passage is mostly 
about 

A. fishing 
trains 

C. people 

D. the moon 



On this queotion you will probably know 
the answer without having a reading 
passage. The word that most nearly 
means "Final* is "B. last.* On your 
answer sheet, fill in the •B* oval for 
question 51. 

On this question you will probably have 
to make a guess. Mark the answer yoa 
choose in the ovals for question 52. 



Read each question carefully because there may be a hint in it about the 
correct answer. Try to answer every question. If you are not certain about 
the answer to a question, pick the answer that seems best and move quickly 
on to the next question. 

When you do finish the questions on a page, turn to the next page and 

do as many pages as you can before you are told to stop. If you finish befor< 

your teacher tells the class to stop, remain quietly j your place. Do not 
turn back to the first test that you took. 

Are there any questions? 

You will have 25 minutes to vork on this test. I will tell you when 
time is up. When you begin this test, the first question you will answer 
will be numbered 53. Remember to nark your answer to this question on your 
answer sheet in the ovals after number 53. 

Now, turn to the next p.i.;;o and bogin to answer the questions. 
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